Skip to content

SCC26: Read WENO stencils directly from v_vf instead of an identical copy - #2092

Draft
AppleChen17 wants to merge 1 commit into
MFlowCode:masterfrom
AppleChen17:feature/weno-read-vf-directly
Draft

AppleChen17 wants to merge 1 commit into
MFlowCode:masterfrom
AppleChen17:feature/weno-read-vf-directly

Conversation

@AppleChen17

Copy link
Copy Markdown

Closes #2091.

Summary

For WENO3/5/7, s_pack_weno_input_arr copies v_vf into v_rs_weno with the same (j, k, l) layout before each reconstruction. This PR makes the WENO3/5/7 reconstruction kernels read v_vf(i)%sf directly. v_rs_weno is kept only for MP-WENO and THINC, which still take it as an argument. Background and the trade-off analysis are in #2091.

On one H200, grind time drops 5.9–6.5% on three example cases with NVHPC 26.3 and 25.9. Outputs are bit-identical to master on our tested cases, and the full GPU and CPU test suites pass.

This PR is from Team NTHU SCC, participating in the SC26 Student Cluster Competition.

Changes

Only src/simulation/m_weno.fpp changes:

  • In the WENO3/5/7 reconstruction loops, v_rs_weno(${SF(...)}$, i) becomes v_vf(i)%sf(${SF(...)}$). The arithmetic and the loop structure are unchanged.
  • v_rs_weno is allocated, packed and deallocated only when mp_weno .or. int_comp > 0, for s_preserve_monotonicity and s_thinc_compression.
  • WENO1 (which already skipped the copy) and MUSCL (which never calls s_weno) are unchanged.

Performance

Grind time (ns/gp/eq/rhs), master → this PR. Each entry is the mean of three interleaved runs within one allocation. All runs use H200, double precision, --case-optimization and the default build flags. Two-GPU runs use two MPI ranks on one node.

Example case NVHPC 26.3, 1 GPU NVHPC 25.9, 1 GPU NVHPC 26.3, 2 GPUs
3D_performance_test (201³, WENO5, 30 steps) 0.41656 → 0.39040 (−6.28%) 0.41475 → 0.39043 (−5.86%) —*
3D_shockdroplet (201×101×101, WENO3, 1000 steps) 0.45402 → 0.42446 (−6.51%) 0.45260 → 0.42298 (−6.54%) 0.28050 → 0.26344 (−6.08%)
2D_shockdroplet --res 4 (4800×1200, WENO3, 565 steps) 0.25042 → 0.23482 (−6.23%) 0.24949 → 0.23476 (−5.90%) 0.13566 → 0.12832 (−5.41%)

Maximum run-to-run spread was 0.81% (NVHPC 26.3, 1 GPU), 0.61% (NVHPC 25.9, 1 GPU), and 2.21% for the two reported 2-GPU cases.

*The 2-GPU 3D_performance_test is omitted: on our node its run-to-run spread (up to 7.8%) was comparable to the effect.

Where the time goes (NVHPC 26.3, Nsight Systems and Nsight Compute):

  • The copy accounted for 7.7% of GPU kernel time in both 3D_performance_test and 2D_shockdroplet. Removing it reduces total kernel time by 6.1% and 6.5%, respectively.
  • WENO5 reconstruction kernels become 5–6% slower due to additional address computation and descriptor accesses, but the eliminated packing kernels more than offset this overhead. L2/DRAM traffic remains within about 1% of master. See Avoid redundant WENO input packing in the common reconstruction path #2091 for details.

Trade-off for MP-WENO / THINC

With mp_weno or int_comp > 0 the copy is still needed, so the small reconstruction overhead is not offset. On 2D_richtmyer_meshkov (WENO5 + MP-WENO, 431 steps), total kernel time is +0.8%, and grind time is +0.45%, within run-to-run variation (up to 2.3%). Avoiding this overhead would require separate reconstruction kernels selected at runtime. We chose not to introduce this duplication for the small observed regression.

Unchanged paths behave as expected: total kernel time is +0.06% for 2D_orszag_tang (WENO1) and +0.03% for 2D_shockdroplet_muscl (MUSCL + THINC), both with bit-identical outputs.

Testing

Baseline: master at 42c44fd.

  • GPU full suite (NVHPC 26.3, OpenACC, H200): 725 passed, 0 failed, 39 skipped. This includes the three mp_weno=T tests and the five WENO5 int_comp tests, which exercise the remaining copy path.
  • CPU full suite (NVHPC 26.3, Intel Xeon Platinum 8480+): 722 passed, 0 failed, 39 skipped.
  • Bit-identical outputs: in our runs, the final-step restart files of every case above match master, including case-optimized builds. No golden files changed.

CPU builds also use the new reads. CPU performance was not benchmarked.

Contribution Policy

All contributions are expected to demonstrate:

  • A clear understanding of the codebase
  • Alignment with product direction
  • Thoughtful reasoning behind changes
  • Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

The WENO3/5/7 reconstruction kernels read their stencil from v_rs_weno,
which s_pack_weno_input_arr fills with an element-wise copy of v_vf
using the same (j, k, l) layout. Read v_vf directly instead, which
removes one read and one write of every reconstructed field per
direction and per RK stage.

v_rs_weno and the pack are still needed by MP-WENO
(s_preserve_monotonicity) and THINC (s_thinc_compression), so they are
allocated, filled and deallocated only when mp_weno or int_comp > 0.
WENO1 and MUSCL do not use s_pack_weno_input_arr and are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Lines of Code

File Lines Diff
src/simulation/m_weno.fpp 1325 +4
Directory Lines Diff
simulation 28530 +4
total 47589 +4

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Claude Code Review

Head SHA: 4fd0f9e

Files changed:

  • 1
  • src/simulation/m_weno.fpp

Findings:
No issues found. Checked in step 4: src/simulation/m_weno.fpp. The context directory was empty, so I read the file from the working tree, which holds the pre-change version, and checked the diff hunks against it. The allocate, pack and deallocate conditions are the same (mp_weno .or. int_comp > 0). The only remaining readers of v_rs_weno are s_preserve_monotonicity (under mp_weno) and s_thinc_compression (under int_comp > 0). The SF coordinates are the unpermuted j,k,l indices that s_pack_weno_input_arr used, so reading v_vf(i)%sf directly with the same offsets stays inside the idwbuff bounds.

@AppleChen17

Copy link
Copy Markdown
Author

The two failing jobs, Github (ubuntu, mpi, reldebug, intel) and Github (ubuntu, mpi, no-debug, intel), look unrelated to this PR. Both fail with an internal compiler error in ifx while building pre_process:

error #5633: **Internal compiler error: segmentation violation signal raised**
compilation aborted for .../fypp/pre_process/m_assign_variables.fpp.f90 (code 3)
compilation aborted for .../fypp/pre_process/m_phase_change.fpp.f90 (code 3)
Error: Failed to build the pre_process target.

The failure happens before simulation is compiled. This PR only changes src/simulation/m_weno.fpp.

The Intel jobs install intel-oneapi-compiler-fortran from the oneAPI apt repo. In this PR's Intel jobs (no-debug, reldebug), apt installed intel-oneapi-compiler-fortran-2026.2 (2026.2.0-863). For comparison, the most recent Test Suite run of #2054 passed every job. Its Intel jobs (no-debug, reldebug) installed intel-oneapi-compiler-fortran-2026.1 (2026.1.1-325).

All other jobs pass, including the GNU jobs on Ubuntu and macOS and the NVHPC 25.5–26.3 CPU and GPU jobs.

@AppleChen17
AppleChen17 marked this pull request as ready for review October 18, 2026 11:30
@AppleChen17
AppleChen17 requested a review from a team as a code owner October 18, 2026 11:30
@AppleChen17
AppleChen17 requested review from danieljvickers and removed request for a team October 18, 2026 11:30
@codecov

codecov Bot commented Oct 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.21053% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.89%. Comparing base (47568a0) to head (4fd0f9e).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
src/simulation/m_weno.fpp 84.21% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2092      +/-   ##
==========================================
- Coverage   61.90%   61.89%   -0.01%     
==========================================
  Files          86       86              
  Lines       22885    22888       +3     
  Branches     3369     3370       +1     
==========================================
+ Hits        14166    14167       +1     
- Misses       6220     6221       +1     
- Partials     2499     2500       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wilfonba

Copy link
Copy Markdown
Collaborator

@AppleChen17 A maintainer will mark this as ready for review when it is suitable. Draft pull requests don't run on self hosted runners, which is preferable when they aren't ready yet.

@AppleChen17
AppleChen17 marked this pull request as draft October 18, 2026 15:36
@AppleChen17

Copy link
Copy Markdown
Author

Thanks for the note, and sorry for the extra runner load. I've converted it back to draft and will leave the ready-for-review decision to the maintainers.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Avoid redundant WENO input packing in the common reconstruction path

2 participants